Coding Theory and Neural Associative Memories with Exponential Pattern Retrieval Capacity

The mid 20th century saw the publication of two pioneering works in the fields of neural networks and coding theory, respectively the work of McCulloch and Pitts in 1943, and the work of Shannon in 1948. The former paved the way for artificial neural networks while the latter introduced the concept of channel coding, which made reliable communication over noisy channels possible. Though seemingly distant, these fields share certain similarities. One example is the neural associative memory, which is a particular class of neural networks capable of memorizing (learning) a set of patterns and recalling them later in the presence of noise, i.e., retrieving the correct memorized pattern from a given noisy version. As such, the neural associative memory problem is very similar to the one faced in communication systems where the goal is to reliably and efficiently retrieve a set of patterns (so called codewords) from noisy versions. More interestingly, the techniques used to implement artificial neural associative memories look very similar to some of the decoding methods used in modern graphbased codes. This makes the pattern retrieval phase in neural associative memories very similar to iterative decoding techniques in modern coding theory. However, despite the similarity of the tasks and techniques employed in both problems, there is a huge gap in terms of efficiency. Using binary codewords of length n, one can construct codes that are capable of reliably transmitting 2rn codewords over a noisy channel, where 0 < r < 1 is the code rate. In current neural associative memories, however, with a network of size n one can only memorize O(n) binary patterns of length n. To be fair, these networks are able to memorize any set of randomly chosen patterns, while codes are carefully constructed. Nevertheless, this generality severely restricts the efficiency of the network. In this thesis, we focus on bridging the performance gap between coding techniques and neural associative memories by exploiting the inherent structure of the input patterns in order to increase the pattern retrieval capacity from O(n) to O(a), where a > 1. Figure 1 illustrates the idea behind our approach; namely, it is much easier to memorize more patterns that have some redundancy like natural scenes in the left panel than to memorize the more random patterns in the right panel. Figure 1: Which one is easier to memorize? Van Gogh’s natural scenes or Picasso’s cubism paintings? More specifically, we focus on memorizing patterns that form a subspace (or more generally, a manifold). The proposed neural network is capable of learning and reliably recalling given patterns when they come from a subspace with dimension k < n of the n-dimensional space of real vectors. In fact, concentrating on redundancies within patterns is a fairly new viewpoint. This point of view is in harmony with coding techniques where one designs codewords with a certain degree of redundancy and then use this redundancy to correct corrupted signals at the receiver’s side. We propose an online learning algorithm to learn the neural graph from examples and recall algorithms that use iterative message passing over the learned graph to eliminate noise during the recall phase. We gradually improve the proposed neural model to achieve the ability to correct a linear number of errors in the recall phase. In the later stages of the thesis, we propose a simple trick to extend the model from linear to nonlinear regimes as well. Finally, we will also show how a neural network with noisy neurons–rather counter-intuitively–achieves a better performance in the recall phase. Finally, it is worth mentioning that (almost) all the MATLAB codes that are used in conducting the simulations mentioned in this thesis are available online at https: //github.com/saloot/NeuralAssociativeMemory.